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ABSTRACT 



This study explored the feasibility of evaluating the 
effectiveness of learning transfer from National Aeronautics and Space 
Administration (NASA) short-term (1 day or less) workshops to the teacher and 
through the teacher to the student, focusing on attitude toward science, 
science-related behavior, and knowledge variables. Participants were 33 
teachers of grades 3-9 from rural, suburban, and urban schools (public, 
private, and home schools) in six states. Teachers had attended 17 NASA 
off-site workshops in which aerospace specialists worked with workshop 
sponsors to plan program content. The objectives of the workshops were to 
enhance teachers' knowledge, experiences, and skills so they could use 
aerospace as a vehicle for improving student learning. Researchers collected 
survey data on teachers’ attitudes, knowledge, and behavior before and after 
the workshops, and they interviewed a sample of workshop participants. They 
also surveyed students about knowledge, attitudes, and behavior before and 
after being taught with NASA resources. Findings support the NASA focus on 
teacher workshops, which are expected to have a ripple effect from 
participants to their students and colleagues. The workshops positively 
affected teachers’ knowledge, attitudes, and behavior. Students and teachers 
supported the NASA focus on teacher workshops. (Contains 24 references.) (SM) 
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This study explored the feasibility of evaluating the effectiveness of learning 
transfer from NASA short-term (1 day or less) workshops to the teacher and through the 
teacher to the student, in the context of attitude (toward science), behavior (science- 
related), and knowledge variables. If results of effectiveness could be found after short- 
term teacher workshops, the assumption of teacher-to-student transfer after longer-term 
teacher workshops (National Research Council, 1994, p. 49) may be extended to short- 
term teacher workshops. But what constitutes effectiveness? 

The literature abounds with studies of the effectiveness of longer-term teacher 
programs (under the aegis of staff development), but is virtually silent about the 
effectiveness of short-term programs, although the short-term NASA programs may be 
considered as staff development. Staff development is an inclusive term that describes an 
array of efforts directed at improving teacher and, ultimately, student effectiveness. It is 
amorphous, with its many forms. 

It can take place in the workplace or in some other environment, it can be required 
or voluntary, it can be offered by an organization or sought independently by an 
individual. Two-hour lectures, three-day conferences and yearlong courses can all 
be considered staff development. (Butler, 1992) 

Within this context, NASA's short-term teacher workshops can be considered a staff 

development effort. Sparks and Loucks-Horsley (1990) proposed five types of staff 

development models. Training is one of the five models, whereby teachers acquire 

knowledge or skills through group instruction, and is the model resembled most closely 

by the short-term NASA workshops, which primarily target inservice teachers. 

According to Bennett (1987), implementation occurs when whatever is targeted in 
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the training, be it increasing knowledge and/or skills or altering an attitude, is transferred 
from the program through the teacher into the classroom and maintained, i.e., observed 
several years later. According to Mathison (1992), we . . think of the staff development 
process as the provision of an inservice experience, the implementation of new 
techniques/skills/ knowledge by teachers, followed by changes in student learning” (p. 
257). Student change, then, not merely implementation, appears to be the indicator of 
successful staff development, yet the staff development literature is lacking research on 
the relationship of short-term workshops to student outcomes. 

Mathi son’s description supported the idea that a teacher workshop is part of a 

system. 

A system is a whole that is both greater than and different from its parts. Indeed, a 
system cannot validly be divided into independent parts as discrete entities of 
inquiry because the effects of the behavior of the parts on the whole depend on 
what is happening to the other parts. (Patton, 1990, p. 79) 

For this study, Bennett’s (1987) definition of implementation was expanded to 

incorporate Mathison’s (1992) and was labeled “systemic implementation.” Therefore, 

systemic implementation occurs when whatever is targeted in the training is transferred 

from the program through the teacher into the classroom as noted by changes in both 

teachers and students. This definition led to the question of what can and should be 

measured to indicate the effectiveness of short-term workshops. 

To explore the first question, the goals and objectives of all NASA teacher 

programs had to be examined. Since NASA does not do education, but supports 

education (B. Bluth, personal communication, August 29, 1998), the intent of the NASA 

workshops is to enhance teachers’ knowledge (content), experiences 
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(confidence/attitude), and skills (competence/behaviors) so they use aerospace 
(aeronautics and space) as a vehicle for improving student effectiveness in science and 
mathematics. Three variables — ^knowledge, attitude, and behavior — ^were identified as 
variables of interest consistent with the goals and objectives of the NASA teacher 
workshops and the definition of systemic implementation. The issue of whether attitude 
and behavior are the same was considered. From past teacher feedback and the 
experience of this researcher, there was evidence that behaviors and attitudes may be 
incompatible and, for this study, were treated as distinct. Therefore, changes in teachers 
and students for these three variables were hypothesized as evidence of workshop 
effectiveness. 

The Study Population 

The study population was distributed throughout six states and represented grade 
levels K-12; rural, suburban, and urban areas; and public, private, religious, and home- 
schools. Participating teachers usually attended the workshops voluntarily and had 
varying degrees of preparation and experiences and varying degrees of administrative 
support for implementation. Participation in the research was ultimately offered to 
workshop participant teachers of Grades 3 through 9 because those grades represented 
most past and current workshop participants. The idea of controlling for demographics 
was discarded to maintain the considerable variability of the population. 

The Sample 

The sample of 33 teachers was drawn from 399 participants who attended 17 
NASA off-site workshops in Ohio and Illinois, with no attempt to control for any 
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variables. The self-selection bias factor was noted as a potential threat to internal validity, 
yet appeared unavoidable in the interest of remaining true to the components of the 
NASA education goals and the simulation of actual workshop conditions, given that the 
usual population of workshop participants is self-selected. 

These workshops were provided by aerospace education specialists from the 
Office of Educational Programs at NASA Glenn (formerly Lewis) Research Center 
located in Cleveland, Ohio, and followed the typical workshop format. Short-term teacher 
workshops vary in length (2 hours to 1 day) and target grade levels and content that meet 
the needs of the sponsoring organization. Workshops may be requested by and for school 
districts, individual schools, the states' regional professional development centers, 
universities, educational conference providers, and others interested in science, 
mathematics, and technology education. The content of NASA as an agency is the 
substance of its education outreach programs. NASA generates knowledge and has within 
its mission the task of disseminating that knowledge. All workshop content related to one 
or more of the four NASA enterprises: Earth Science, Aerospace Technology, Human 
Exploration and Development of Space, and Space Science. 

The aerospace specialists work with the workshop sponsor to plan the content 
appropriate for the audience and for the time constraints. Whereas the content of the 
workshops may vary, all specialists address the national science and mathematics 
standards and model inquiry-based science and cooperative learning. Despite the 
theoretical consistency, each specialist has a different presentation style and a different 
degree of proficiency in each of the content areas. Respectively, each teacher takes back 
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something different to the classroom, depending on his/her interests, proficiency, needs, 
and the influence of the presenting specialist. 

Joyce and Showers (1981) examined effective staff development programs. After 
identifying common elements, they developed five components they posited as increasing 
the chances for successful transfer of the workshop knowledge and skills by the teacher 
into the classroom. The NASA specialists incorporate four of the components into the 
typical workshop: theory, modeling, practice, and feedback. The fifth component, 
coaching, is not a formal option, although the specialists do respond to occasional 
inquiries from past participants. 

For this study, 15 of the 17 workshops were of 1 full day’s duration; two were 
shorter. Fourteen utilized the same content and format and were conducted by the same 
specialist. The others involved that specialist and two others. The researcher attended 
nine of the workshops in order to recruit volunteers. At the other eight workshops, the 
presenter used the established script and protocol to recruit additional volunteers. 

The sample of 823 student participants was separated into two levels, depending 
on the level of the student forms their teacher selected as appropriate to that group. 
Ultimately, results were based on an n of 256 for students who used the level one forms 
and an n of 567 for students who used the level two forms. From demographic data 
collected on the Teacher Survey I form, both samples were found to be representative of 
the teacher and student populations in the two states on the following variables: numbers 
of public and private school teachers, highest degree earned and number of years of full- 
time teaching experience for public school teachers (state data not available for private 
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school teachers), and the distribution of student enrollment by race. Additional 
demographics of the teachers, their schools, and districts reflected the heterogeneity of 
the sample of teachers and students. 

The Teacher Subsamnle of Interviewees 
A subsample of 10, 30% of the sample, was deemed a suitable number to be 

interviewed. Everyone in the sample had given consent for an interview. In order to 
create a subsample that would be representative of the sample, specific demographics 
were designated as important to match proportionately: school location (urban, other 
urban, suburban, rural), race/ethnicity, age range, sex, school level, number of years 
teaching, highest degree, and school type. The subsample was considered to be 
representative of the full sample. For confirmation, the means of several other 
demographics of the subsample were compared to the full sample. Representation was 
considered confirmed. 



Design of the Study 

The design of the study was nonexperimental. This was appropriate since there 
was no attempt to manipulate the independent variables and randomization was not 
present (Pedhazur & Schmelkin, 1991, p. 304). Multiple measures of attitude, behavior, 
and knowledge were used. 

The intent/objectives of NASA workshops are to enhance teachers' knowledge 
(content), experiences (confidence/attitude), and skills (competence/behavior) so they use 
aerospace (aeronautics and space) as a vehicle for improving student effectiveness. 
Therefore, in this study, systemic implementation occurred when that which was targeted 
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in the training was transferred from the program through the teacher into the classroom as 
evidenced by changes in teachers' and students' knowledge of, attitude toward, and 
behaviors related to science. Assuming that attitude is related to confidence and 
competence is related to behavior, in this study the same distinction was given to 
confidence and competence as was given to attitude and behavior, i.e., they were 
considered separately. Henceforth, these terms will be used interchangeably — content 
with knowledge, confidence with attitude, and competence with behavior. 

Research Question and Hypotheses 

The research question asked if there were changes in terms of knowledge, attitude, 
and behavior of teachers and students after the interventions. This question translated to 
six different sets of hypotheses. The null hypotheses stated that there were no differences 
between population average scores for teachers and students on the pre- and postmeasures 
of knowledge, attitude, and behavior. The alternative hypotheses stated that the 
population average scores on the premeasures were less than the population average 
scores on the post measures. 

Key Definitions 

Systemic implementation is a term coined for this study to define an expansion of 
Bennett’s (1987) definition which incorporated Mathison’s (1992) description of the staff 
development process. Systemic implementation is the transfer of the workshop 
objectives to the teacher and through the teacher to the student as evidenced by changes 
in both teachers and students. This definition became the basis for the research question 
which investigated the existence of such changes. 
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The independent variables in the study were (a) the workshop and (b) whatever 
the teacher taught from the workshop. It was recognized that everyone takes away from 
workshops different ideas and interests. In order to make the research project realistic, 
participants were free to choose how they wanted to use the resources, e.g., as a lesson, as 
a unit, integrated. The teachers reported on the final survey the resources that were used 
and how they were used. 

The identification of the three dependent variables for teachers and the same three 
dependent variables for students generated the six operational definitions shown below, 
which were intended to provide answers to the research question. 

1 . Enhancement of teacher knowledge : increase in knowledge from pre- to 
postmeasure in the workshop setting 

2. Enhancement of teacher confidence : improved attitude toward teaching 
science from self-report before the workshop and again after using the NASA 
resources 

3 . Enhancement of teacher competence : increased science-related behaviors 
reported before and after using the NASA resources 

4. Change in .student knowledge : increase/improvement in pre- and postmeasures 
of knowledge before and after their teachers used the NASA resources 

5. Change in student attitude : pre- and postself-reports of attitude before and 
after their teachers used the NASA resources 

6. Change in student behavior : pre- and postself-reports of science-related 
behaviors before and after their teachers used the NASA resources, along with 
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teacher observation and reports. 

Data Sources and Collection 

Attitude Variable 

Multiple measures of teacher attitude were included because attitude is a complex 
construct. Self-report was selected as a method for this and other measures, because its 
efficacy is theory-based. For example, according to the learning theory of Carl Rogers 
(1997), one of the qualities of experiential learning is self-evaluation as the principal 
method of assessing progress or success. Malcolm Knowles’ (1997) theory of adult 
learning includes self-evaluation as one of the most useful strategies for instruction. 

The preworkshop form asked participants to rate their expectations/ enthusiasm 
for the workshop. Participants also had the opportunity to describe their expectations. A 
quick and early perusal of expectations was expected to assist the presenter in ensuring 
that expectations were met and in clarifying the workshop objectives, particularly for 
cases where the expectations would not match the objectives. 

After the workshop, participants again rated themselves. The questions provided 
indicators of attitude toward the workshop, the presenter, using knowledge and skills 
gained fi'om the workshop, attending another NASA workshop, and recommending the 
workshop to their colleagues. The Teacher Survey II form, completed after teaching with 
the NASA resources, provided additional attitude indicators by asking participants’ 
comfort level teaching with the resources and the type of follow-up they desired. Finally, 
both the postworkshop form and the Teacher Survey II form had an open-ended question 
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With support from the literature (Ofir, Reddy, and Bechtel 1987; Emmerson, 

1988; Kearney as cited in Rubin et al., 1994), a semantic differential attitude 
questionnaire was chosen from the quantitative tradition, . . because it is a highly 
controlled method of attitude assessment . . . provides a measure of the direction and 
intensity of the respondent's evaluation. . . (LeSourd, 1984). 

An existing measure of motivation was adapted for the teachers and for students 
in Grrades 6 through 9. The Student Motivation Scale (SMS) is an established instrument 
that employs the semantic differential scale and was adapted for this study. It was 
included on the teachers' preworkshop form as a measure of attitude toward teaching 
science. It was included again as a postassessment on the Teacher Survey II forms that 
teachers completed after they used the NASA resources with their students. The SMS was 
also used as the level two forms for students. The researcher to provide similar feedback 
from students who would use the level one form created a corresponding elementary level 
Likert scale. Recall that teachers selected the level of forms appropriate for their students. 
Open-ended questions on the student attitude pre- and postmeasures provided additional 
information on this variable. Students were asked about their career goals on both the pre- 
and postmeasures and were asked on the postmeasure to write what they liked about 
studying science. Teachers, on their Teacher Survey II forms, reported whether student 
interest was higher for the NASA topics than for other science topics. In addition, during 
telephone interviews some teachers discussed student attitude/interest. 

Behavior Variable 

As with the attitude variable, several measures of behavior were used. The 
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instrument to assess behaviors associated with interest in science, e.g., watching TV 
science programs, was developed by the researcher from a list collected in a poll of 35 
teachers who were asked how they know, other than from standardized test scores, that 
their students are interested in science. From that list, instruments were developed for the 
teachers and for both levels of students. 

For teachers, the majority of indicators came from the Teacher Survey I and II 
forms. On the initial Teacher Survey I form, which the volunteers submitted prior to 
teaching the NASA content, teachers used a 6-point Likert scale to describe themselves. 
“About You as a Teacher” asked them to rate behaviors such as watch science-related TV 
shows and subscribe/read science-related professional journals. On that form teachers 
also estimated similar behaviors of their students. “About Your Students” asked teachers 
to rate the percentage of their students on a scale of 0-25%, 26-50%, 51-75%, 76-100% 
on behaviors such as watch science programs (not assigned) on TV and read science- 
related materials (not assigned). A second survey of teacher behaviors and reported 
student behaviors was completed by teachers after teaching the material from the 
workshop as part of the Teacher Survey II form, which also included evaluative questions 
related to the workshop and teaching experiences. One additional indicator came from the 
postworkshop form and asked when, how, and what NASA resources they expected to 
use from the workshop. The Teacher Survey II form addressed when, how and what they 
actually used. 

For students, variations of the self-report behavior instrument were designed for 
the two levels of student forms and were given by the teacher as pre- and postmeasures. 
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These included an open-ended question by which they could say more about what they do 
related to science. Additional data about the students came from teacher comments on 
the Teacher Survey II form and during the telephone interviews. 

Knowledge Variable 

Because the workshops covered different knowledge areas, four different pre- and 

postmeasures were developed for each of the targeted workshops, i.e., one for each of the 

three pilot workshops and one for all the Illinois workshops, although three questions 

were consistent overall. Questions for the teacher knowledge tests were developed 

jointly by the researcher and the aerospace education specialists who conducted the 

workshops to ensure that the knowledge measured met the objectives of the topic(s). 

These were included with the pre- and postworkshop forms. For the knowledge portion 

of the postworkshop forms, participants were instructed as follows: 

These are the same questions you answered before the workshop. There is no 
need to repeat what you wrote earlier. Simply add anything that you learned from 
the workshop that you did not include before. 

Not only did this process save time, critical in short workshops, but it also provided a 

dichotomous measure of knowledge (gain yes or no, i.e., additional items were or were 

not added). The teachers’ content questions were adapted by the researcher as pre- and 

postmeasures for use at an education level of Grades 3 through 5 and for use at an 

education level of Grades 6 through 9. After rater training, a panel of three rated teacher 

gains and a different panel of six rated student gains. 

In addition to the gain indicator, the same three knowledge questions for both 

teachers and students provided quantitative data. Two were simple recall questions 
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asking about NASA missions and objects in the solar system. The third question required 
problem solving by asking what is needed to live in space. A simple scoring system was 
used to award points to the premeasure question and then to the corresponding 
postmeasure question. The researcher scored the teacher forms. A graduate assistant 
scored the student forms. With input from the specialists, it was decided to have teachers 
and students give a self-rating of knowledge of the topic on both pre- and postmeasures, 
which provided a third measure of the variable. 

Interrater reliability . Two methods were used for determining interrater reliability 
of the teacher gain measure. After training, three raters each rated the same five practice 
tests. After each of the first three practices, the results were discussed. Since all raters 
were in agreement for practice tests four and five, the remaining tests were scored without 
further discussion. With the Average Percent Agreement method, agreement was 97%. 
Because of such a high percentage, the second method was used in order to take out 
chance occurrence, a limitation of the Average Percent Agreement method. GWISE 
(Dimitrov, 1997) is designed to provide kappa coefficients of agreement among multiple 
raters for nominal scales. The coefficients provide agreement over and above the 
agreement that may occur by chance. The ratings for each question were consistent 
between any two raters and all three raters. The coefficients were high. The reliability of 
the raters could be trusted. 

Because of the large student N (823), not all raters of the student forms were able 
to rate all test sets as was done for the teacher sets, a requirement of the GWISE method. 
Interrater reliability was ascertained only by the Average Percent Agreement method. 
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After training, six raters each rated the same 10 practice tests. After each practice, the 
results were discussed and the process was further refined. Raters, by consensus, 
increased the list of acceptable responses. Then 30 test sets were scored by all without 
discussion. Since total Average Percent Agreement was 84%, the panel decided that the 
reliability of the raters could be trusted and the remaining tests were split among the 
raters for scoring. Nevertheless, the GWISE method is preferred and is recommended if 
an actual evaluation results from this study. 

Telephone Interviews 

A sample of workshop participants who gave their permission (on the consent 
form) was interviewed by telephone. Interviews were audiotaped. Some participants 
asked to hear the tape before authorizing use of the data in this study. The audiotapes 
were destroyed after the results of the study were written. 

Interview questions included some structured and some open-ended questions. 
For example, indicators of use in the classroom of knowledge and skills gained in the 
workshop were questions that asked which activities, strategies, and materials from the 
workshop were used. An indicator of the success of the thematic approach taught in the 
workshop was whether participants integrated aerospace education into curricular areas 
other than science. Indicators of more general outcomes were descriptions of use of the 
workshop gains outside of their classrooms, such as conducting a workshop for other 
teachers. 
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Both quantitative and qualitative analyses were selected to provide triangulation, 
the objectivity of quantitative methods and the rich contribution of the qualitative. 
Quantitative analyses of data from instruments for the three variables were done for 
teachers before and after the workshop and before and after teaching and for students 
before and after their teachers used the NASA resources. Quantitative analyses were done 
at the .05 level of significance and included descriptives, dependent t tests, repeated 
measures, post hoes, profile plots, group profiles, factor analysis, canonical correlation, 
and two methods of interrater reliability (previously discussed). 

Qualitative data sources for teachers for the variables of attitude, behavior, and 
knowledge were taped telephone interviews with nine teachers and notes from a 
telephone interview with one teacher. Source data for the same variables related to 
students came from several open-ended questions on the student instruments and from 
teachers' reports of their students. Data were analyzed by the Grounded Theory 
procedure, which is often referred to in the literature as "the constant comparative method 
of analysis" (Glaser & Strauss as cited in Strauss & Corbin, 1990, p. 62). This method 
was selected because it provided a complement to the quantitative methods; it is highly 
regarded and well accepted in the field of qualitative data analysis. With this procedure, 
data were examined individually for themes and then across the data for similarities and 
differences, or patterns, which led to categorization and conceptualization. 




Results 

Of the original 80 teachers who submitted Consent Forms, 58 provided the initial 
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Teacher Survey I form, thereby indicating to the researcher intent to complete the 
research project. Of these 58 teachers, considered the actual number of volunteers, 46 
(12% of the 399 total workshop participants; 79% of the volunteers) reported that they 
actually taught what they learned in the workshop. Of those 46, 33 (8% of the total 
workshop participants; 57% of the volunteers) provided dociunentation by completing the 
Teacher Smwey II form and submitting student pre- and postdata for their students. 

The number of workshop participants who did not volunteer for this project yet 
did implement what was learned was unknown. Confidence intervals were estimated 
using a chart found in Glass and Hopkins (1984, p. 280), which provided .95 confidence 
limits for the population proportion, given the sample proportion. In this case, the sample 
proportion was the documented 33, i.e., 8% (p = .8) of total workshop attendees (399) 
who transferred what they learned into the classroom. From the chart, it was determined 
that the lower limit of the .95 confidence interval was .06 and the upper limit was .13. 
Therefore, one can be 95% confident that if the entire population of workshop 
participants had been surveyed, rather than the 80 who submitted Consent Forms, 
between 6% and 13% would have transferred what they learned into the classroom. The 
sample proportion of 8% appeared representative of the population of workshop 
participants. 

For teachers, statistically significant differences were found in behavior and 
knowledge. Statistically significant differences were found in behavior and knowledge 
for all students and for the attitude variable for students who used the level two forms. 
While results of quantitative measures can be summarized in the previous two sentences. 
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Teachers 

Attitude 

For the attitude variable for teachers, the first instrument was a semantic 
differential to determine attitude toward teaching science before the workshop and after 
teaching. Group results were plotted on the instrument itself The profiles were nearly 
identical, allowing for the conclusion that there was no change in attitude toward teaching 
science. To confirm the results of the profiles and test the null hypothesis, a dependent t 
test compared average pre- and postscores for individuals and indicated no significant 
difference in attitude from pre- to postmeasure. This was not unexpected and is 
consistent with what is commonly known with regard to workshop attendance. The fact 
that 94% of the teachers voluntarily attended the workshop explained the high positive 
attitude toward teaching science and was also consistent with expectations. 

A dependent t test compared attitude toward the workshop from premeasure 
(expectations/enthusiasm) to postmeasure (satisfaction with the workshop). There was no 
significant difference from pre- to postmeasure. This was consistent with the average 
ratings (maximiun score of 4) for workshop expectations/ enthusiasm (3.7) on the 
premeasure and satisfaction (3.8) on the postmeasure. It appeared that expectations were 
met. The presenter received an average score of 3.9 and all would recommend the 
workshop to colleagues and take another NASA workshop. Of 87 desired types of 
follow-up, 14% wanted high level content, 33% wanted materials, and 26% wanted 
teaching strategies. None said they wanted no follow-up. An average score of 3.8 for 
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using in the classroom knowledge and skills gained was not unexpected since 93% of 
participants already taught science. 

The results are incomplete without the contribution of the qualitative data. 
Although there was no significant statistical difference in the attitude of teachers, there 
were reports of inspiration, increased confidence, and increased knowledge that led to 
behavior changes. For example, one teacher said she had taken several university courses 
in astronomy, because of her interest in the topic, but did not feel comfortable teaching 
astronomy to her 5th-grade students until the NASA workshop. The workshop provided 
her with the materials, activities, and language to do so, thereby increasing her confidence 
(attitude). Some teachers simply noted feeling more comfortable with teaching the subject 
after the workshop. Other teacher comments reported high student interest, helpful 
information, easy activities, increased confidence. All of the teachers had positive things 
to say about their experiences with the NASA workshop and about teaching with the 
NASA resources. This was expected, since teachers volunteered to participate in the 
study. 

Behavior 

The first measure consisted of 14 questions which were rated on a scale of 1 to 6 
(1 = low, 6 = high) to best describe the behaviors that teachers do. One item was 
discarded after noting comments from several teachers that they knew how to and desired 
to use the Internet yet had no access. Therefore, the item “use Internet” would not be a 
valid indicator of actual Internet behavior. All other items presented options that were 




realistic and were retained. 
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Through principal components analysis three components were identified and 
accounted for 62% of the variance on the premeasure. Three components were also 
identified for the postmeasure and accounted for 65% of the variance. Statistically these 
were nice groupings, but conceptually they did not make sense; the items were combined 
and averaged and a t test compared the average scores of individuals on the 13 items. The 
2 value was significant. 

The next measure of behavior examined what participants said they would use 
from the workshop and what they actually used. Those who said they would use “all” or 
“most all” were not included in the count. Many materials were distributed and discussed 
at the workshops. Only some could be used at the workshop because of time constraints. 

It appeared that the number of activities/resources that were actually used exceeded the 
number forecast. This was unexpected, given that teachers in general have said they 
usually use only what they actually “practice” in the workshop because they have no time 
to “practice” otherwise and do not like to do activities they have not tried themselves. It 
is suggested that the degree of confidence (attitude) affects the selection of resources 
used. 

The interviews generated additional information that supported the quantitative 
results. A few of the teachers had already covered the space topics earlier in the year, yet 
were able to revisit them for the research project because these NASA resources were 
current and new to their students. Several teachers borrowed Moon Rocks from the 
Educator Resource Center. Two teachers said they did more with aeronautics and flight 
this time. Frequently lauded were the posters and new activities. Electronic resources 
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were also mentioned. Some teachers commented that they planned to continue and do 
more in the upcoming school year. 

Knowledge 

The first measure used was the self-rating by participants of their knowledge of 
aerospace before and immediately after the workshop. A dependent t test compared 
scores on the pre- and postmeasures. The significant p value was not unexpected, since 
the postmeasure was given merely 6 hours after the premeasure, unlike the tests of the 
null hypotheses for attitude and behavior where for some participants as much as 5 
months had elapsed between the pre- and postmeasures. 

The second measure of knowledge used a combination of three questions on the 
workshop pre- and postmeasures. These questions were selected because they were asked 
of both teachers and students and were easily quantified, while the other questions were 
open-ended. The researcher assigned 1 point for each correct response per question. The 
questions were coded as NASA (names of NASA missions), NEED (what’s needed to 
live in space), and OBJ (objects in our solar system). To test the null hypothesis, repeated 
measures, with time as the within-subjects factor, compared pre- and postscores of 
individuals by question while accounting for the correlations among the three questions. 
The Wilks’ Lamda statistic was significant (F (3,24) = 18.52, p < .0005), indicating a 
difference between pre- and postmeasures for at least one of the questions. Univariate 
tests showed significance for all of the questions. One possible explanation may be the 
short time between the two administrations. 

Next, the model was upgraded by successively including one of the following 
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between-subjects factors: sex, race, highest degree, undergraduate major, graduate major, 
school type (public, private), and school site (urban inner-city, urban-other, suburban, 
rural). A significant main effect was found only for the between-subjects factor school 
site (F (2,24) = 3.65, p < .05) on the variable OBJ, indicating a significant difference 
between the average OBJ score for at least two of three levels of school site (only one 
school was categorized as urban-other and was excluded from the analysis by the 
program). Post hoes (Tukey a) reported a significant difference for pre- and postmeasure 
scores between rural and urban inner-city teachers. This was an unexpected finding, 
prompting further examination. A graph of the means defined an ordinal interaction 
between the OBJ question with teachers from both urban inner-city schools and rural 
schools. Once the topic of that question was examined, the fact that interaction occurred 
with only one of the questions was not surprising. The content of the OBJ question is 
available to the general public through the media, electronic resources, books, toys, 
games, museums, science sites, and so forth, educational resources that are more likely 
accessible to suburban settings. Supporting this premise was the fact that there was no 
gain from pre- to postmeasure for suburban teachers for the OBJ question. 

The next measure of teacher knowledge utilized a rating panel of three and all the 
knowledge questions. The panel compared each premeasure question with the 
corresponding postmeasure question and assigned either a “yes” or “no” to the 
postmeasure question to indicate whether or not there was a gain in knowledge. Then 
they noted the percentage of questions for which they had given a “yes.” No one had a 
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zero gain; 30% showed a gain on seven or nine of the 10 questions; 70% had a gain on 
two to five questions. 

It was also important to examine the percentage improvement for each individual 
question to inform any redesign of the workshop. Conclusions need to be perceived in the 
context of the questions and participant responses. For example, the low percentage (7%) 
of improvement noted for Question 7 was the result of a straightforward question (“The 

first ‘A’ in NASA stands for ”)• Since most participants answered correctly on 

the premeasure, there was low gain on the postmeasure. However, the 22% for question 
9 (“Give an example of how ‘drag’ helps us and how it hinders us”) was the result of a 
difficult question. The data in combination with the questions could be used for 
normative evaluation leading to improvements in the workshop design and delivery. 

The quantitative analyses showed significant differences from pretest to posttest 
on the self-knowledge rating and on each of the three knowledge questions. The 
interviews, which were conducted 3 to 6 months after the workshops, supported the 
perception by participants of knowledge gained, which was a highlight for many of the 
teachers. 




Students 

Attitude 

The first measure examined attitude toward science before and after being taught 
with the NASA resources. For students who used the level one forms, the mean pre- and 
postscores for each individual question were examined by question and showed 
minuscule changes in the means for all questions from pre- to postmeasure. For students 
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who used the level two forms, group pre- and postprofiles were examined. The profiles 
were nearly identical. It appears that there was no change on this measure in student 
attitude relative to the NASA resources. 

To test the null hypothesis, a dependent t test compared the average pre- and 
postscores for individuals on the first measure, attitude toward science. For students who 
used the level one forms, the t value showed no significant difference from pre- to 
postmeasure. This was not unexpected and was consistent with what is commonly 
known with regard to attitude changes following an intervention. For students who used 
the level two forms, the p value was significant. This result was not expected and was 
inconsistent with commonly held beliefs about attitude changes following an 
intervention. 

As with the teachers, the qualitative data provided additional information. 

Students were asked about their career goals. Although nothing conclusive could be 
ascertained, it appeared that the exposure to the NASA resources might have caused some 
students to think positively about careers related to aerospace, thereby signifying a 
possible change in attitude. From teacher reports, the materials and activities stimulated 
interest in their students. The newness of the resources and the fact they were easy to 
incorporate also added to student interest. Some of the teachers reported high interest 
when they used the materials with children in their families and with one’s church group. 
It is safe to speculate that a coimection exists between the attitudes of these teachers and 
their students’ attitudes. 
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Behavior 



The first measure consisted of a self-report instrument for each of the two levels 
of forms. Students who used the level one form rated 12 behaviors, e.g., “I watch science 
programs on TV,” as “Never,” “Sometimes,” or “Often.” An additional open-ended 
question asked them to write what else they liked to do in science and math. Students 
who used the level two form rated 13 behaviors on a scale of 1 to 6 (1 = low, 6 = high). 
An additional open-ended question asked them to write what else they do that is related to 
science. The item “use Internet” was discarded from the student analysis just as it was 
from the teacher analysis as an invalid indicator of actual Internet behavior. Students 
might know how to and desire to use the Internet yet have no access. The final analysis 
included 1 1 items from the level one form and 12 items from the level two form. 

Principal components analysis with Promax oblique rotation provided a structure 
matrix for each level form with the criterion of retaining only those components whose 
eigenvalues were greater than 1. For students who used level one forms, four 
components were identified for the premeasure and accounted for 57% of the variance. 
After learning from the NASA resources, there was a regrouping of some of the variables, 
indicating a positive change. Three components were identified for the postmeasure and 
accounted for 53% of the variance. 

Statistically these were good groupings, but conceptually they did not make sense. 
For example, “school-related behaviors” had been theorized as a label for one factor. On 
the premeasure, “like to come to school,” “like to learn about careers in science and 
math,” and “like to solve problems” loaded together. On the postmeasure, “like to come 
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to school” loaded with “use the library for fun things,” “like to solve problems,” and “like 
to read about science for fun.” On the premeasure, “complete my homework” loaded 
with “visit science places outside of school,” while on the postmeasure it loaded with 
“like presents like science books, games, or models,” “like to talk about science things,” 
and “like to watch science programs on TV.” Speculating that students in the earlier 
grades used the level one forms, school attendance and completion of homework were 
probably integral to the total learning experience. Since it made sense conceptually to 
group all the items for this level, the items were combined and averaged. 

For the students who used the level two form, the same two factors emerged for 
both the pre- and postmeasures, accounting for 48% and 50%, respectively, of the 
variance. Homework and attendance loaded together both times as the only two items on 
the second factor, confirming the theorized label of “school-related.” The other 10 items 
loaded together both times on the first factor, although in a different order, and confirmed 
the theorized label of “science-related.” 

To test the null hypothesis for level one data, a t test compared the average scores 
of individuals on the 1 1 items. To test the null hypothesis for level two data, a t test 
compared the average scores for individuals on the 10 science-related items; an additional 
t test compared the average scores for individuals on the two school-related behaviors. 

For students who used the level one forms; the p value was significant. For students who 
used level two forms, the p value was significant for factor one, but not for factor two. 

To examine which relations of variables on the pre- and postmeasures remained 
stable across the treatment, canonical correlation analysis was used. Although the 
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requirement of 20:1 (Stevens, 1992) was not met with an n of 256 for level one forms, it 
was appropriate to use the standardized scores from the canonical correlation analysis to 
attempt to reduce the variables. The n (567) for level two forms did meet the first ratio 
criterion of 20:1 but not the second. However, since the factor analysis had shown two 
definite factors, it was decided to interpret the first two canonical correlations for level 
two data. 

All canonical correlations were significant for data from both levels between the 
sets of questions on the pre- and the post postmeasures. For level one data, the first 
canonical variate represented the relationship between variable 1 1 (“I like to come to 
school.”) on both pre- and post postmeasures. It was significant and of magnitude .83. 

R, = .83 (X = .02, f (121) = 804.65, p < .01) 

Both of these canonical variates had the highest canonical loadings as well as the highest 
standardized canonical coefficients. 

For level two data, the first canonical variate represented the relationship between 
items 1 (“Watch science programs [not assigned] on TV”), 4 (“Talk about science-related 
events”), and 10 (“Read science-related materials”) on both pre- and posttests. It was 
significant and of magnitude .84. 

R, = .84 (X = .00, x' (144) = 2417.45, p < .01) 

It is interesting to note that all 10 items previously grouped as one factor (science-related) 
had high canonical loadings, ranging from .46 to .76 on both the pre- and postmeasures 
for the first canonical variate. As such, all 10 items could be interpreted as contributing 
together to that first canonical correlation. The second canonical variate represented the 
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relationship between the two items previously grouped as factor two (school-related) for 
both pre- and postmeasures. It was significant and of magnitude .78. 

R 2 = .78 (X = .01, x' (121) = 1879.46, p < .01) 

An examination of the z scores showed there were some redundant variables. To 
make the instrument more parsimonious and for stability fi-om pre- to post postmeasures, 
the number of variables was reduced by eliminating those with the smallest z scores, i.e., 
those below . 1 . For level one data the premeasure items were reduced fi"om 1 1 to 5; the 
postmeasure items were reduced fi"om 1 1 to 6. For level two data, the premeasure items 
were reduced fi"om 12 to 6; the postmeasures items were reduced fi"om 12 to 8. 

The last measure of student self-report was an open-ended item. For students who 
used level one forms, the item simply said, “Write what else you like to do in science and 
math.” For students who used level two forms, the directions instructed them to “Write 
what else you do that's related to science.” Recall that the questions were different 
because of the expectations of delivery by workshop participants. Teachers of students 
using level one forms were more likely to teach both science and mathematics, thereby 
providing the possibility of using NASA resources in either class. Students who used 
level two forms most likely received the NASA materials in a science class, since the 
overwhelming majority of workshop participants teach science. 

For students who used level one forms, the majority of the responses mentioned 
liking multiplication, problem solving, projects/experiments, computers/Intemet, and 
planets/space on both the pre- and postmeasures. For students who used level two forms, 
the majority of responses on both pre- and postmeasures included experiments. 
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computers, reading about science, science fairs, space/stars, field trips, and models. 
Differences in percentages between pre- and postmeasures in all categories for both levels 
were so slight that no inferences could be made. 

Teachers provided further insight into student behaviors. Thirteen teachers made 
a total of 19 comments about noticeable positive student behaviors related to use of the 
NASA resources. None reported negative behaviors. Of the comments, nine addressed 
the involvement of the students in the hands-on activities, two addressed the use of teams 
and cooperative learning, two provided competitions with the activities, two reported 
science-related behaviors of students of teachers with whom resources were shared, and 
three addressed volimtary additional time and work by students. One teacher, who would 
have the same class the following year, spoke of the request by her students to do more 
next year. Clearly, the richness of the qualitative data expanded the results of the study. 
Knowledge 

The first measure used was the self-rating by students of their knowledge of flight and 
space (level one forms) and aerospace (level two forms) before and after working with the 
NASA resources. To test the null hypothesis, a dependent t test compared scores on the 
pre- and postmeasures; there were significant p values for both levels. The results were 
particularly noteworthy in that the time lapse between pre- and postmeasures varied 
considerably from class to class. Although teachers did not report the actual amoimt of 
time spent, the number and complexity of resources used could provide an estimate. 

The second measure of knowledge used a combination of three questions on the 
premeasure and the same three questions on the postmeasure. Recall that these questions 
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were selected because they were asked of both teachers and students and were easily 
quantified. Following the same procedure used with the teacher data, a graduate assistant 
assigned one point for each correct response per question. The questions were coded as 
NASA (names of NASA missions), NEED (what’s needed to live in space), and OBJ 
(objects in our solar system). 

To test the null hypothesis, repeated measures, with time as the within-subjects 
factor, compared pre- and postscores of all students (N = 823) by question while 
accounting for the correlations among the three questions. The Wilks’ Lamda statistic 
was significant (F (3, 701) = 327.65, p < .05), indicating a difference between pre and 
postmeasures for at least one of the questions. Univariate tests for all students showed 
significance for all the questions. Next, the model was upgraded by including form level 
as the between-subjects factor. A significant main effect was found with all three 
questions. Post hoes (Tukey a) were not permitted by the program since there were fewer 
than three groups. To examine which level contributed to the significant findings, t tests 
for each form level compared average scores for each question. The t statistic was 
significant for each question at each form level. 

The next measure of student knowledge utilized a rating panel of six and all the 
knowledge questions. Each panelist rated part of the groups by comparing each 
premeasure question with the corresponding postmeasure question and assigning either a 
“yes” or “no” to the postmeasure question to indicate whether or not there was a gain in 
knowledge. For each individual, they also noted the percentage of questions for which 
they had given a “yes.” For most questions, there was a substantial gain. However, 
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conclusions need to be perceived in the context of the questions with participant 
responses. A low percentage of gain might result from most participants answering 
correctly on the pretest. Information from the data in combination with the questions 
could be used for normative evaluation leading to improvements in teaching. 

Overall, only 16 students who used the level one form (8%) and 11 students who used the 
level two form (3%) showed no gain on the instrument. Those students were scattered 
throughout the sample and not from single classes. Only two students who used the level 
two form gained 100%. The majority of students from both form levels showed a gain of 
more than 60%. 

Comments by teachers enhanced the understanding of this variable. Past 
comments to the researcher by some teachers that the NASA materials and activities 
would not work with their students (e.g., Special Education, primary grades) were 
disproved by several of the teachers in the research project. The teacher of 7th- and 8th- 
grade Special Education students reported that her students’ scores “soared.” A teacher 
who integrated the NASA resources into reading because her school was on probation, 
reported that her fourth and fifth graders learned a lot, which helped when they got ready 
to take the science part of the test. In addition to the pre- and postmeasures provided for 
the research project, teachers reported other evidence of student gains: their observations, 
the use of peer coaching, and cooperative learning. 

As with student attitude and student behavior, it appears that knowledge was 
increased for these students after their teachers used the NASA resources. This finding 
might be evidence of transfer of learning from these teachers to their students, with 
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notable changes in both groups. 

Discussion 

Findings support the NASA focus on teacher workshops, which are expected to 
have a “ripple” effect from participants to their students and colleagues. Each act of the 
workshop or act of teaching related back to the objectives of the workshop, demonstrating 
reciprocity and recursiveness. A positive attitude (increased confidence) appears to have 
led to increased competence (behaviors). Knowledge might have been the catalyst or the 
outcome. 

NASA workshops appeared to have affected the attitudes of these teachers, 
thereby enhancing their perception of their effectiveness. A teacher’s positive attitude is 
not only critical in developing student interest, but may be considered critical to the 
effectiveness of teaching. “One teacher characteristic that has been consistently related to 
student achievement, but which is not systematically measured is teacher efficacy” (Huitt, 
1997). Teachers have long considered reaching just one student in a class an indication 
of success. It is suggested that reaching only one teacher at a workshop should also be 
counted a success, especially when one considers the “ripple effect” — ^the numbers of 
students and colleagues who will be touched by that one teacher. 

Of the measures of behavior, descriptives from questions related to extensions of 
the workshop (NASA resources used, integration, sharing with colleagues, sharing with 
other classes) appeared to support the dependent t test which showed a significant 
difference in behaviors related to teaching science. Some of the activities extended 
beyond the individual classrooms to involve other classes or the entire school. All these 
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teachers reported doing some things they had not done before. The evidenced behaviors 
support the NASA focus on teacher workshops. It is suggested that the statistically 
significant increase in behaviors reflected a practically significant increase in confidence 
(attitude) of the sample. 

Several measures of teacher knowledge provided expected and unexpected 
findings. The significant difference fi'om pre- and postmeasure on the self-ratings was 
supported by the significant difference fi'om pre- to postmeasure on the three knowledge 
questions scored by the researcher. Investigation of gains scored by a rating panel 
provided information relevant to each question and to the workshop design. 

One of the knowledge questions, which showed a significant difference fi'om pre- 
to postmeasure asked participants to name NASA missions. The workshop provided the 
teachers with much new knowledge about current NASA missions. In attendance during 
half the number of workshops, the researcher observed participants asking many 
questions related to the Mars Pathfinder and Cassini missions, MIR, and the forthcoming 
International Space Station. Participants also appeared eager for additional sources with 
which to stay current, yet appreciated having the opportunity to interface with a NASA 
person. It appears that increased knowledge influenced these teachers' comfort levels and 
thus improved their attitudes. As noted earlier, a positive attitude about science appears to 
have influenced these teachers' behaviors. It seems safe to speculate that increased 
knowledge was critical to their sense of improved effectiveness. 

In examining student attitudes, an open-ended question asked what students liked 
about studying the space topic. For students who used the level one forms, 81% reported 
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positively and 19% negatively, compared to the students who used the level two forms, 
where 65% reported positively and 35% reported negatively. Speculating that students 
who used the level two form were in higher grades than those who used the level one 
forms, this finding was consistent with what is accepted in the field, that students are 
interested in science in the early grades and tend to lose interest as they move up. Also 
consistent was the fact that the t test for the students who used the level one forms 
revealed no significant difference related to attitude toward science. On the other hand, 
despite the 35% negativity for students who used the level two forms, the statistically 
significant t value related to attitude toward science suggested that a change in attitude 
could be effected with this intervention. This reinforces NASA’s efforts to target middle 
school teachers. 

Corroborating the positive aspects, 91% of the teachers reported higher student 
interest for the NASA topic than for other science topics. In general open-ended 
questions, several teachers commented on the high student interest. These results 
supported the value of the NASA educational resources for generating student interest 
and positive attitude. Not surprising was that teacher attitude appeared to influence 
student attitude. 

Results of the various analyses of student behavior, reported by the students 
themselves and their teachers, support the NASA focus on teacher workshops. These 
results provided evidence of the predicted “ripple effect” fi-om participants to their 
students and colleagues. Note that significant differences were found in student self- 
reports of science-related behaviors for students who used both level one and two forms. 
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The fact that there were no significant differences from teacher reports of the same 
behaviors may be due to a limitation of the study which is discussed later. Perhaps, 
though, self-assessment is a more accurate measure, for individuals may continue to 
model the way they perceive themselves as behaving. Recall also that 13 teachers made a 
total of 19 comments about noticeable positive student behaviors related to use of the 
NASA resources and none reported negative behaviors. For the behavior variable, as with 
student attitude, materials and activities played a critical role. The teacher of 7th- and 8th- 
grade Special Education students reported that her students did things that people just 
didn’t think they would be able to do and that she knew it came from using the NASA 
materials. 

An unexpected finding was related to the research project itself One teacher was 
surprised to learn that there was a real lack of experiences with science outside of school. 

Also unexpected were the reports by some teachers of fewer science-related behaviors 
on the postmeasure. Discrepancies were attributed by some to “guesses” on the 
premeasure followed by greater attention to the actual frequencies of the behaviors. The 
project served to inform those teachers. For this group of teachers, it appears that 
positive teacher attitude and the NASA resources contributed to the perceived increases 
in science-related behaviors of their students. 

An examination of student results on the three knowledge questions raised an 
interesting issue. For the question NASA (name NASA missions), students who used the 
level two forms scored higher on both the pre- and postmeasures with a larger gap on the 
postmeasure, indicating that these students gained more than the students who used the 
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level one forms. On the OBJ question (name objects in the solar system), students who 
used the level two forms again scored higher on both and showed a slightly larger gap on 
the postmeasure. For the NEED question (what is needed to live in space), however, the 
students who used the level one forms scored higher on both the pre- and postmeasures, 
with the gap on the postmeasure only slightly smaller than that on the pretest. 

These results may be explained by noting the type of questions and speculating 
that the older students used the level two forms and the younger students used the level 
one forms. Scores on both the NASA and OBJ questions would most likely be higher for 
individuals who have more background in science, i.e, older students who not only had 
more science education but who were probably more aware of world news and science 
events. Describing what is needed to live in space required problem solving and critical 
thinking skills. Higher scores by the younger students might simply be a result of 
younger students not yet being afraid to be “wrong.” On the other hand, it might be 
evidence of the success of the current reform efforts resulting from the National Science 
Standards and the National Mathematics Standards . 

In summary, several measures of knowledge provided expected and unexpected 
findings. The significant difference from pre- to postmeasure on the self-ratings was 
supported by the significant difference from pre- to postmeasure on the three knowledge 
questions scored by the researcher. Investigation of gains scored by a rating panel 
provided information relevant to each question. These results also supported the 
theorized “ripple effect” of the NASA teacher workshop. 
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Examining the Study: Limitations and Lessons Learned 

Following the idea of formative evaluation, the study was examined for 
limitations and ways to improve it. First, recall that the method used for the attitude and 
behavior variables was self-report, which, for this study, was a practical choice and 
supported by the fact that it is theory-based (Rogers, 1997; Knowles, 1997). 

Nevertheless, self-report may be considered a limitation. 

Second, although none of the workshops was held at a NASA center, and 
although the NASA workshop format may be similar to workshops from other providers, 
the simple fact of a NASA workshop might be a limitation of the study, for people tend to 
associate excellence with NASA. Participants may experience a sense of elitism, which 
may influence them to use the materials and share with colleagues. The NASA 
association lends credibility to what they take back to their schools. Teachers previously 
reluctant to teach certain concepts may do so using NASA materials. Despite the 
limitation, this study may inform the designs of other workshop providers. 

Examination of the instruments revealed opportimities for improvement. First, 
the instruments themselves could be refined. Based on responses, several questions on the 
Teacher I Survey should be rewritten. From the way the questions were worded, there 
was no way to determine whether the NASA workshop where teachers were recruited 
was included in the teacher report of the number of science workshops or the number of 
NASA workshops attended. When the subsample was questioned, some teachers 
included NASA workshops in the number of science workshops attended; others did not. 

From questions teachers asked the researcher during some of the workshops, it was noted 
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that some schools did not differentiate between “professional days” and “workshop days” 
while others did. The data were not usable from these questions. 

Although choices for the levels of the schools were listed on the form as K-4, 5-8, 
and 9-12, to be consistent with designations in the national science and mathematics 
standards, 39% of the respondents selected “other,” which required the creation of new 
variables based on the grades these teachers taught. Surprisingly, the “other” designations 
were not just in rural areas as one might expect. The high percentage that answered 
“other” should serve to inform agencies that collect such data, including NASA whose 
online tracking system also uses these designations in an effort to be consistent with the 
education community. 

Also on the first survey was the instrument for teacher report of science-related 
behaviors of students. The instrument should be revised to avoid the wide ranges. For 
example, a teacher would check the range of 26% to 50% on the pretest to indicate that 
30% of the students talked in class about science topics in the news. If that number 
increased to 50%, the teacher would again check the range of 26% to 50%, camouflaging 
the gain. 

Finally, examination of the protocol also revealed areas that could be improved. 
Because of the brevity of the workshops, there were some teachers who were unable to 
complete the Teacher Survey I form at the workshop itself, but mailed it later. Recall that 
the instrument for teacher report of student science-related behaviors was part of that 
survey. Teachers who completed the form during the workshop had to estimate their 
students’ behaviors while those who mailed them back later were able to more accurately 
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calculate the percentages that exhibited each behavior. It is suggested that the teacher 
report of student science-related behaviors be completed when teachers do not have to 
estimate. It is also recommended that the protocol be revised to include a procedure for 
reminders, such as is done for mail-in surveys. Perhaps earlier reminder calls might have 
resulted in a larger sample. An additional suggestion is to include questions in the final 
survey that would inform the protocol. 

Implications 

This study provided pieces of information not previously available. It showed 
that it is feasible to evaluate NASA’s short-term teacher workshops. It appears that the 
overall directional research hypotheses of the study were confirmed for these samples. In 
other words, NASA short-term teacher workshops led to transfer and increased teacher 
and student effectiveness in knowledge, attitude, and behavior. The findings support 
NASA emphases on teacher workshops, middle schools, and rural and urban areas. 
Although the samples were small, participant demographic characteristics were consistent 
with the average populations of the two states, lending credibility to the results. 

It is up to Agency management to decide if this study is sufficient evidence for an 
assumption that the short-term workshops are effective. Building upon lessons learned 
from this study, similar results from studies in additional states could provide a stronger 
basis of evidence for the effectiveness of the NASA short-term teacher workshops and the 
assumption of student-to-teacher transfer. Continued feedback via NASA’s current 
online Education Evaluation System will continuously inform the design of future 
workshops. Meanwhile, “Be satisfied with the evidence if absolute proof isn’t possible to 
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attain” (Kirkpatrick, 1996, p. 57). 

The study may also have implications for the training community, because it 
achieved the four levels of the esteemed Kirkpatrick model, first introduced in 1959, that 
is widely used to evaluate organizational training. Put simply, the four levels are reaction, 
learning, behavior, and results. According to Kirkpatrick (1996), “Evaluation becomes 
more difficult, complicated, and expensive as it progresses from level 1 to level 4 — and 
more important and meaningful” (p. 56). He went on to say that, although the four levels 
are not always used, they are necessary for a meaningful evaluation. 

Most importantly, however, this study has implications for the education 
community. Concerns about our public educational system are evidenced by the growth 
of home schooling, charter schools, reform strategies and other increasing efforts to better 
educate the nation’s youth. As teacher accountability grows, staff development grows 
even more important. One of the National Education Goals in the Goals 2000: Educate 
America Act (1994, p. 42109) states 

The Nation’s teaching force will have access to programs for the continued 
improvement of their professional skills and the opportunity to acquire the 
knowledge and skills needed to instruct and prepare all American students for the 
next century, (p. 42109) 

As professionals, educators must be concerned about the structure of that staff 
development and ensure that whatever the structure, it contributes to teacher effectiveness 




and student improvement. 
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